UNIVERSAL SUBSPACES FOR COMPACT LIE GROUPS 



JINPENG AN AND DRAGOMIR Z. BOKOVIC 



Abstract. For a representation of a connected compact Lie group G in a 
finite dimensional real vector space XX and a subspace V of XX, invariant under 
a maximal torus of G, we obtain a sufficient condition for V to meet all G- 
orbits in XX, which is also necessary in certain cases. The proof makes use of the 
cohomology of flag manifolds and the invariant theory of Weyl groups. Then 
we apply our condition to the conjugation representations of U(n), Sp(n), 
and SO(n) in the space ofnxn matrices over C, H, and R, respectively. In 
particular, we obtain an interesting generalization of Schur's triangularization 
theorem. 



1. Introduction 

Let p be a representation of a connected compact Lie group G in a finite di- 
mensional real vector space 11. We say that a linear subspace V c U is universal 
if every G-orbit meets V. Examples of universal subspaces occur in many places. 
For instance, when p is irreducible and nontrivial, it is easy to show (see [12] ) that 
every hyperplane of U is universal. For the adjoint representation of G in its Lie 
algebra g, it is well known that every Cartan subalgebra is universal. If we consider 
the complexified adjoint representation of G in the complcxification gc of g, then 
every Borel subalgebra of gc is universal, and if moreover G is semisimple, then the 
sum of all root spaces in gc is also universal ([8]). Questions of similar nature have 
been recently studied in the context of representation theory of algebraic groups, 
see e.g. HH2D]. 

In this paper we consider the case where the stabilizer of V in G contains a 
maximal torus T of G. Our main result (Theorem 14. 2\ gives a sufficient condition 
for the universality of V, which is easy to check in concrete examples. With an 
additional restriction on V this condition becomes also necessary (see Theorem 

E3). 

Our method goes as follows. Consider the trivial vector bundle Eu = G/T x U 
over the flag manifold G/T. Since V is T-invariant, there is a well-defined subbundlc 
Ey of Eu whose fiber at gT is p(g)(V). Every vector b£U, viewed as a constant 
section of Eu , induces a section s u of the quotient bundle Eu / E-y ■ It is easy to see 
that the G-orbit of u meets V if and only if s u has a zero. So V is universal if and 
only if s u has a zero for every u € U. A sufficient condition for this is that Eu/Ey 
is orientable and has a nonzero Euler class. If we choose a T-invariant subspace W 
of XL complementary to V and construct similarly the subbundle Eyy of Eu, then 
Eyj = Eu/Ey. It is easy to see that Ew is always orientable and is equivalent to 
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G Xt W, the associated bundle induced by the T-action on W. By using a theorem 
of Borel on the cohomology of G/T and the invariant theory of Weyl groups, we 
can compute explicitly the Euler class e(E-w) of Eyj. 

Let X(T) be the character group of T, and let iS be the graded algebra of real 
polynomial functions on the Lie algebra of T. The map from X(T) to H 2 (G/T, R) 
sending \ £ X(T) to the first Chern class of the complex line bundle G x x C 
induces a homomorphism ip : S — > H* (G/T, R) of graded algebras, provided we 
double the degree in S. Borel's theorem asserts that <p is surjective and its kernel 
is the ideal I of S generated by the basic generators of W^-invariant polynomials, 
where W — Nc(T)/T is the Weyl group. If we assume that the above subspace W 
of 1L does not contain nonzero T-invariant vectors, then W can be decomposed as 
a direct sum © i=1 Wj of 2-dimensional T-irreducible subspaces. The T-action on 
every is equivalent to the real T- module associated to some \% € X(T), and then 
G Xj-Wj is equivalent to the real vector bundle associated to G x Xi C. So e(Eyj) is 
equal to the product of the first Chern classes of G x Xi C, i.e., Ylf =1 <f(Xi)- We refer 
to the homogeneous polynomial /y = Xi as the characteristic polynomial of V. 
So e(i?w) 7^ if and only if /y (/ T, and we obtain the desired sufficient condition 
for the universality of V. In general, the question of whether fy is in X can be 
resolved using Grobner bases. 

An easy dimension argument shows that if V is universal, then dimV > dim IX — 
dim G/T . The case of equality is of particular interest (indeed, for many representa- 
tions, a universal subspace V whose universality can be detected using our method 
can be shrunk to a smaller universal subspace for which the equality holds). In this 
case, the rank of Eyj is equal to dim G/T, and /y has degree m = \ dim G/T. By 
invariant theory of Weyl groups, a homogeneous polynomial of degree m is in T if 
and only if it is orthogonal to the fundamental harmonic polynomial /o (see (|2.5p ) 
with respect to a IF-invariant inner product on S. So if we define the characteristic 
number Cy of V as the inner product of /y and fo (up to a constant factor, which 
will be chosen in a way suitable for the computation of intersection numbers), then 
fy ^ T if and only if Cy ^ 0, which gives a simpler equivalent sufficient condition 
for the universality of V. Moreover, if we assume that the intersection indexes of 
the above section s u and the zero section so of Eyj are all equal whenever s u is 
transversal to sq, then V is universal if and only if Cy ^ 0. 

In concrete examples that we shall consider, the characteristic polynomials and 
numbers are easy to compute. We shall do this for the conjugation representations 
of U(n), Sp(n), and SO(n) in the spaces M(n, F) ofnxn matrices over F = C, H, 
and R, respectively, for subspaces of M(n, F) determined by zero patterns. A zero 
pattern is a subset / of {(i,j)\i,j S {1, . . . , n}, i ^ j} of cardinality n(n— l)/2. For 
F = C or HI and a zero pattern /, we denote by M/(n, F) the subspace of M(n,F) 
consisting of matrices whose (i, j)-th entry is whenever (i, j) G I. We shall apply 
the above result to M/(n, F) and obtain a sufficient condition for the universality 
of M/(n,F) expressed as a simple property of I. In particular, we show that if 
/ contains exactly one of and (j, i) for i ^ j (we call such I a simple zero 
pattern), then M/(n,F) is universal. Similar results are also obtained for F = R. 
Note that if Iq = < i < j < n}, M/ (n, C) is the subspace of all complex 

lower triangular matrices, its universality is known as Schur's triangularization 
theorem [25j . So our result gives a genuine generalization of Schur's theorem. If 
F = C and / is a bitriangular zero pattern (defined in Section[5]), the above sufficient 
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condition for the universality of M/(n, C) is also necessary, and the number of flags 
in U(n)/U(l) n sending a generic matrix into M/(n, C) can be computed directly 
from I. 

The problem of universality of subspaces of M(n, C) determined by zero patterns 
has been raised by researchers in linear algebra. For instance, the universality 
of M/(4, C) when I = {(1, 3), (1, 4), (2, 4), (3, 1), (4, 1), (4, 2)} has been studied by 
several authors. It was shown by Pati |23j that it is universal. A simpler proof was 
given later in [5] where it is also shown that the number of flags sending a generic 
matrix A 6 M(4, C) into Mj(n, C) is equal to 12. This zero pattern is bitriangular 
and so both results follow immediately from Theorem 15. 21 Another simple example 
is the problem of universality of M/(3, C) where I = {(1, 3), (2, 1), (3, 2)}. This was 
raised a few years ago [7] but remained unsettled until now. In our terminology I is 
a simple zero pattern, so M/(3,C) is universal (see Theorem 15 .61 and the paragraph 
preceding it) . For further information along this direction see e.g. [SJ [TS] . 

The arrangement of this paper is as follows. In Section 2 we shall recall some 
preliminaries on Euler classes, intersection indexes, cohomology of flag manifolds, 
and invariant theory of Weyl groups. The definitions of the characteristic poly- 
nomial and the characteristic number of a subspace and their properties will be 
given in Section 3. The main results will be proved in Section 4. In Section 5 we 
shall study the universality of subspaces determined by zero patterns, generalizing 
Schur's triangularization theorem. Then in Section 6 we shall briefly discuss the 
possibility of generalizing our method and propose a question. 

The first author would like to thank Jiu-Kang Yu for valuable conversations. 

2. Preliminaries 

In this section we recall some basic facts about Euler classes of oriented vector 
bundles, cohomology of the flag manifold G/T, and invariant theory of Weyl groups. 
Throughout this paper, we denote by G a connected compact Lie group and by T 
a maximal torus of G. 

2.1. Euler classes and intersection indexes. Let M be a connected compact 
smooth n-dimensional manifold, and let E be a smooth real oriented vector bundle 
over M of rank r. It is well known that if the Euler class e(E) € H r {M, Z) of 
E is nonzero, then every smooth section of E has a zero. If E = E\ E% as 
oriented vector bundles, then e(E) = e(i?i)e(£ , 2). If E admits a complex structure 
compatible with the orientation, then the top Chern class Ct op (E) of E is equal to 
e(E) (see [22]). 

Now we assume that M is oriented and r = n. Let sq be the zero section of 
E. Then for any smooth section s of E, we have e(E)([M]) — I(s,sq), where 
[M] £ H n (M,Z) is the fundamental class of M and I(s,So) is the intersection 
number of s and sq. Let Z(s) — {x G M\s(x) = 0} be the zero locus of s. We 
write s t\\ x so if s is transversal to sq at the point x € Z(s), and s (tl So if s &\ x sq 
for all x £ Z(s). If s rh sq, then Z(s) is finite and I(s,sq) = J2 x ez(s) m da;(s), 
where ind^s) = ±1 is the intersection index of s and So at x. Assume that E 
is a subbundle of a trivial bundle M x IX, where U is a real vector space, and let 
us view s as a map M — > 11. For x € Z(s), we can compute ind^s) as follows. 
Since s &\ x sq, the tangent map (ds) x : T X M — » Toll = IX is injective and has image 
E x , the fiber of E at x. If we view (ds) x as an invertible map T X M — * E x , then 
ind^s) = sgn((ds) x ). (Here the sign sgn(A) of an invertible linear map A : V\ — ► V2 
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between real oriented vector spaces refers to the sign of the determinant of the 
matrix of A with respect to any ordered bases of V\ and V2 compatible with the 
orientations.) 

We remark that if E is the tangent bundle TM of M, then e(TM)([M]) is the 
Euler characteristic of M. 

In this paper, M will almost exclusively be the flag manifold G/T. We shall 
need some facts about the cohomology of G/T, which we now recall. 

2.2. Cohomology of G/T. Consider the maximal torus T of the connected com- 
pact Lie group G. Let X(T) be the character group of T. Every character \ € X(T), 
viewed as a representation of T in C, induces a complex line bundle L x = G x x C 
over G/T, where the total space of L x is the space of T-orbits in G x C with the 
T-actioni- (g,z) = (gt- 1 ,x{t)z). Let ci{L x ) e H 2 (G/T,Z) be the first Chern class 
of L x . Then the map sending \ to c\(L x ) induces a homomorphism of abelian 
groups 

X(T) -> H 2 (G/T,Z). (2.1) 

Note that X(T) can be identified with a lattice in t*, the dual of the Lie algebra t of 
T , by using differentials. It will be more convenient to work with cohomology with 
real coefficients. (As H*(G/T,Z) is torsion free ([21]. Chapter 6, Theorem 4.21), 
no information is lost when Z is replaced by K.) So we consider the linear map 

t* ~ X(T) ® z R -> H 2 (G/T,R) = H 2 (G/T, Z) ® z K (2.2) 

induced by the homomorphism (|2.ip . 

Let iS be the symmetric algebra of t*, identified with the graded algebra of 
real polynomial functions on t. Let H*(G/T,M) be the graded algebra of real 
cohomology of G/T. The homomorphism (|2.2p induces an algebra homomorphism 

<p:S -> H*(G/T,R) (2.3) 

which doubles the degree. Let W — Nq{T)/T be the Weyl group. Then W 
acts naturally on t, t* and S. Let J be the graded algebra of ^-invariants in S, 
let Sd be the homogeneous component of degree d in S, and let Jd = 3 C\ Sd 
and = d>ll 7d- Let X = SJ+ be the ideal of S generated by J+ and set 
l d =lC\S d . 

Theorem 2.1 (Borel [2], Theorem 26.1). T/ie above homomorphism ip is surjective 
and ker(tys) = X. Consequently, H*(G/T,M) = 5/1 as graded algebras, provided we 
double the degrees in S /T. 

Although Borel did not mention Chern classes explicitly in his theorem in [2], 
his presentation is equivalent to ours. Chern classes are explicitly used in the 
approaches to Borel's Theorem in [TJ [Til EE HH [27] . Our construction of the 
homomorphism ip follows from [19], Chapter VI, Theorem 2.1 and [27] (see also 
[TT] . Section 10.2, Proposition 3). This homomorphism can be constructed also by 
using classifying spaces. Let BK denote the classifying space of a group K. Then 
(p is equal to the composition of the Chern- Weil isomorphism S — > H* (BT,M.) 
and the homomorphism H*(BT, M) -> H*(G/T,R) induced by the fiber bundle 
G/T — > ST — > BG (see [9] [2T]). 

At last, we recall the well known fact that the Euler characteristic x{G/T) of 
G/T is equal to \W\ (see e.g. [21], Chapter 5, Theorem 3.14). 
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2.3. Invariant theory of Weyl groups. We have seen above that H*(G/T,M) 
can be described by VF-invariants in the symmetric algebra S. To understand 
H*(G/T,M) more closely, we now recall some basic facts about invariant theory of 
the Weyl group W. For details see [M l \T7 \ [26] . 

For H £ t, the partial derivative dnf of / £ S is defined by [djjf)(H') = 
^\ t — f(H' + tH). Let (•, •) be a VT-invariant inner product on t. Then there is a 
VF-equivariant isomorphism t : t* — > t defined by the identity (i(a),H) — a(H), 
under which the map H t— > dn extends to an isomorphism / i— > df of S onto the 
algebra of differential operators on t with constant coefficients. The bilinear form 
B on S defined by B(f,g) = (dfg)(0) is a VT-invariant inner product on S. If 
ai,...,ak,0i,...,Pk £ t*, then 

k 

B(a 1 ---a k ,f3 1 ---f3 k ) = flWaO.^Ci))), (2-4) 

where Sfc is the symmetric group on k letters. It is easily seen that df is the adjoint 
operator of the multiplication operator by /, that is, B(g, fh) = B(dfg, h) for any 

f,g,h£ S. 

A polynomial / <E S is W -harmonic if d g f = for all g E J+. Let H denote the 
space of W-harmonic polynomials, and let Hd = Hfl Sd- Let $ C X(T) C t* be 
the root system of G, <I> + a system of positive roots, and m = |$ + | = i dim G/T. 
Some basic facts that we are going to use are collected in the following theorem. 

Theorem 2.2. (1) There are r = dimt basic homogeneous W -invariant polyno- 
mials F\,...,F r which are algebraically independent such that the subalgebra of S 
generated by F± , . . . , F r is J , and the ideal of S generated by F\, . . . , F r isX = <SJ7+. 

(2) The degrees di = deg-Fi are determined by W , Yli=i di = 1^1; S[=i(^ — 1) = 

TO. 

(3) S = I ®7i orthogonally with respect to B, and the Poincare polynomial ofS/X 
is given by Y^=ii ( ^ m '^-d)i d = n[=i(l +£ + ••• + t di ^ 1 ). In particular, 7id = for 
d > to and dim TL m = 1 . 

(4) The polynomial 

fo= n a ( 2 - 5 ) 

is in Tim, and 7i = {dffo\f S S}. 

(5) For f 6 S, f € X if and only if df fo = 0. In particular, for f € S m we have 
f € X m if and only if B(f, f ) = 0. 

(6) f £ S is W -skew if and only if f £ f§3 ■ (Here f is called W -skew if w ■ f = 
sgn(w)/ for every w £ W.) 

We refer to the polynomial fo £ Ti. m given in (|2.5p as the fundamental harmonic 
polynomial. Given / £ S m , it is important to know when / £ X m . Theorem l2.2l (5) 
provides a characterization using /q and the inner product B. Although it is not 
difficult to compute B using (|2.4p . the following generalization of Theorem 12.21 (5) 
will be more convenient. 

Proposition 2.3. X m = f^ with respect to any W -invariant inner product on S m . 

Proof. By Theorem l2.2l f6l . every M^-skew polynomial in S m is a multiple of fo. So 
for the natural representation of W in S m , the isotypic component of the irreducible 
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representation w i— > sgn(w) is R/o- Hence the orthogonal complement of R/o is 
independent of the choice of the W- invariant inner product on S m . □ 



In view of Proposition 12.31 the fundamental harmonic polynomial /o plays an 
important role. For later reference, we give below explicit expressions for fo and a 
VF-invariant inner product on S for classical groups. Before doing this, we define a 
polynomial Ao in R[xi, . . . , x n ] as 

n 

Ao(x) = n ^ - xj) = j2 ^wn^S' ( 2 - 6 ) 

l<i<j'<?i ctGS„ i=l 

where x — (x\, . . . , x n ). This polynomial will appear in all cases of classical groups. 

Example 2.1. Let G = U(n), let T = U(l) n = {t = diag(*i, . . . , i„)|*i € f7(l)} 
be the standard maximal torus and t ~ {x — diag(y— Txi, . . . , \f— Tx n )|xj S R}. 
Then 5 can be identified with the polynomial algebra R[xi , . . . , x n ] , and a character 
x(t) = t™ 1 ■ ■ ■ <™" with the linear form m\X\ + • • • + m„i„. The root system is 
given by 

$ = {±{x l - Xj)\l <i <j <n}, 

and $ + can be chosen as 

< & + = {xi - Xj\l < i < j < n}. 

So in this case 

/o(x) = A (x). 

We have deg/o = |$ + | = n(n — l)/2 and W = S n . There is a unique VF-invariant 
inner product on S for which the monomials x™ 1 • • • x™ 71 form an othornormal basis. 
The basic generators F\ , . . . , F n of X can be chosen as the elementary symmetric 
polynomials in xi, . . . , x n . □ 

Example 2.2. Let G = Sp(n). T — U(l) n is a maximal torus and t = {x = 
diag(V~ l^i, ■ ■ ■ , \/— lx n )|xi € R}. So S = R[xi, . . . , x n ). The root system is given 

by 

$ = {±Xi ± Xj\l < i < j < n} U {±2xi|l < i < n}, 
and $ + can be chosen as 

$ + = {x t ± x 3 |l < z < j < n} U {2xj|l < i < n}. 

So 

n 

fo(x)=2 n X a (xl,...,x 2 n )Y[x i . 

i=l 

We have deg/ = |$ + | = n 2 and VF = (Z/2Z) n x S'n. The VF-invariant inner 
product on S can be chosen as in Example 12.11 The basic generators Fi , . . . , F n of 
I can be chosen as the elementary symmetric polynomials in xf , . . . , x„ . □ 

Example 2.3. Let G = SO(2n). T = SO(2) n is a maximal torus and t = 

x = diag (7° i , . . . , "o )) }• So S - R ^ ' ■ ' ' The root 

system is given by 

$ = {±Xi ± Xj\l < i < j < n}, 

and $ + can be chosen as 

<J> + = {x t ± xj\l < i < j < n}. 
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So 

f (x) = Xo(xl, . . .,a£). 
We have deg/ = |$+| = n(n-l) and W ^ (Z/2Z) n ~ 1 x S n - The VF-invariant inner 



product on S can be chosen as in Example 12.11 The basic generators F% , . . , , F n of 
I can be chosen as the first n — 1 elementary symmetric polynomials in ajf , . . . , x^ 
andnLi^i- 1=1 



Example 2.4. Let G = SO(2n + 1). T = SO(2) n x {1} is a maximal torus and 

t={, = diag((_° i * 1 ),-,(_° n X o),o)}. So 5^ 4 The 

root system is given by 

$ = {±Xt ± Xj\l < i < j < n} U {±Xj|l < i < rt}, 
and $ + can be chosen as 

<£+ = {xi ±afj-|l < i < j < n} U {Xi\l <i <n}. 

So 

n 

fo(x) = X (xl,...,xl)Y[xi. 

1=1 

We have deg/ = |$+| = n 2 and = (Z/2Z) n x S„. The ^-invariant inner 
product on 5 can be chosen as in Example 12.11 The basic generators Fx , . . . , F n of 
X can be chosen to be the same as in Example 12.21 □ 



3. Characteristic polynomials of subspaces 

Let p be a representation of G in a finite dimensional real vector space II, and 
let U T be the subspace of T-invariant vectors in 11. Let V C U be a T-invariant 
subspace containing U T . We shall now define the characteristic polynomial fy of 
V (up to a sign which depends on the orientation of 11/ V). 

Let W be a T-invariant subspace of U complementary to V. Every real irre- 
ducible representation of T is either trivial or 2-dimensional. In the latter case, it 
is equivalent to the the real T-module associated to some nontrivial character \ of 
T. Since U T C V, there is no trivial subrepresentation of T in W. So W can be 
decomposed as a direct sum 

d . 
W = 0W„ d=-dimW, (3.1) 

i=l 

of 2-dimensional T-irreducible subspaces. Note that the real T-modules associated 
to x and — \ are equivalent. To associate to W, a unique Xi € ^(X)\{0}, we choose 
and fix an orientation on W^. Then there is a unique Xi € X(T) \ {0} for which 
there exists a T-equivariant orientation preserving isomorphism Oi : (R 2 , Xi) ~> W*, 
where (R 2 ,Xi) is the real T-module associated to x% with the standard orientation. 
The orientations of the Wi's induce an orientation on W~ U/V. It is easy to see 
that the product of the Xi' s depends only on the induced orientation on W, and 
changes sign if the orientation is reversed. 
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Deflnition 3.1. Given the orientation on W = U/V as above, wc define the char- 
acteristic polynomial fy G Sd of V by 

d 

h = Y[xt 

i=i 

The equivalence en above induces a complex structure Ji on W;, which is inde- 
pendent of the choice of Oi and is determined by the orientation and the T-action 
on Wi. The direct sum 

d 

Jw = J (3.2) 

i=i 

is a complex structure on W compatible with the orientation. For any nonzero 
vectors Wi € Wj, the ordered basis 

{wi,Jwwi,...,w d ,JwWd} (3.3) 
of W is compatible with the orientation. 

Remark 3.1. If XL admits a complex structure such that the G-action is complex 
linear, and if V C U is a complex subspace, then we can choose W to be a complex 
subspace. Thus W has the complex decomposition 

W= W x , 

where W x is the x-isotypic component of W. As U T C V, we have Wo = 0. Each 
W x can be further decomposed as a direct sum of complex lines, which serve as the 
Wi's in (|3.1[) . If the orientation on each of these complex lines is chosen to be the 
one induced by the original complex structure, then 

h= n x dimcw *, 

xeX(T) 

and Jw coincides with the original complex structure on W. 

We now consider the trivial real vector bundle Eu = G/T x U over the flag man- 
ifold G/T. For any T-invariant subspace L of U, there is a well-defined subbundle 
En of Eu whose fiber at gT is p(g)(L). In particular, V and W are T-invariant, 
and so we have the subbundles Ey and Eyj- Since U = p(g)(V) p(g)(W) for every 
g 6 G, we have Eu — Ey Eyj . The T-action on W preserves the orientation. So 
the orientation on W induces an orientation on Eyg = Eu/Ey, and we can consider 
the Euler class e(E w ) = e(E u /E v ) G H 2d (G/T,R). Since J w is T-invariant, it 
induces a complex structure on So we can also consider the top Chern class 
Ctop(-E'w) G H 2d (G/T,M.). Since the orientation and the complex structure are 
compatible, e(i?w) = Ct op (£'w). 

Theorem 3.1. We have e(Eu/Ey) = ip(fy), where (p is the homomorphism (|2.3p . 

Proof. We view Eyg as a complex vector bundle and prove that Ct op (T'w) = v(/v)- 
Let Li be the subbundle of Eu whose fiber at gT is p(g)(Wi). The complex structure 
Ji on "Wi induces a complex structure on Li. By (|3.1[) . Eyj = ® i=1 Li as complex 
vector bundles. So we have 

d 

Cto P (Ew) ^YlciiL,), (3.4) 

j=i 
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where ci(Tj) is the first Chern class of Li. For \ G X(T) let L x = G x x C be the 
complex line bundle over G/T induced by \- We claim that Li = L Xi as complex 
line bundles. Indeed, given any nonzero vector ua G Wj, the map G x C — > Li 
defined by 

(g,z) (gT,p(g)(zwij) 

induces an equivalence between L Xi and L^. So by the definition of tp, we have 

ci(ij) = ci(i x< ) = <p(Xi)- (3.5) 
From (|3.4p and (J5TSJ) we deduce that 

d d 
ctop(-Ew) = n^( x ^ = ^(Q Xl ) = ^(/ v )- 

2=1 8=1 

□ 

Set m = \ dim G/T. We shall be mainly interested in the case d = m. In this 
case, it will be useful to know the explicit value of e(Eu/Ey) ([G/T]), where [G/T] 6 
i?2m(G/T, Z) is the fundamental class of G/T with respect to some orientation on 
G/T. We fix the orientation on G/T as follows. Let g and t be the Lie algebras 
of G and T, respectively, and let $ C X(T) be the root system of G and $ + = 
{cti, . . . , a m } a system of positive roots. Let gc = tc © © Q6 $ 9a be the root space 
decomposition of the complexification of g, and set m = gD (0 a£$ g a ). Then there 
exist X a G g Q such that 

{X ai — X_ ai , \f—\(X ai + X_ ai ), . . . , X Qm — X-ctm > V— l(^ Qm + X_ arn )} (3.6) 

is a basis of m. For a G $ + , let m Q = span{X Q — y^T(X a + X^ a )}, and let 
J a be the complex structure on m a defined by 

Ja(X a — X- a ) = y/— l(X a + X-a), J Q (V '~l(X a + X- a )) = — [X a — X— a ). 

Then m = (B ae $+ tn Q , and J m = J Q is a complex structure on m, which 

induces an orientation on m compatible with the ordered basis (|3.6[) . For g G G, 
define the map r s : m — > G/T by 

r g (X)= 3 e*T. (3.7) 

Then the differential {dr g )o : m — ► T s t(G/T) is invertible and induces an orientation 
on T 9 t(G/T), which depends only on gT. This induces an orientation on G/T. 

Definition 3.2. If d — m, we define the characteristic number Cy of V by 

r (fv, fo) , W | 
[Jo, Jo) 

where W is the Weyl group, (•, •) is a VF-invariant inner product on S m , and fo is 
the fundamental harmonic polynomial defined in 



Proposition 3.2. Gy is independent of the choice of (•,■). 

Proof. The linear functional 5 m — > M sending / to takes the value \W\ 

at fo and vanishes on /q = I m . □ 

We shall prove that, with respect to the above orientation on G/T, e(EHv)([G/T]) = 
Cy. We first prove a special case. Let T^ be the subbundle of the trivial bundle 
G/T x g corresponding to the T-invariant subspace m. 
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Lemma 3.3. For the adjoint representation of G in q, we have ft = fa and 
e(E m )([G/T]) = C t = \W\. 

Proof. With respect to the complex structure J a , the T-action on m a is complex 
and is equivalent to a. So ft = Y[ a e<s>+ a = f° an< ^ ^t = |W|. 

To finish the proof, we note that the orientation preserving map 

(dT g ) o Ad^)- 1 : (E m ) gT = Ad( 5 )(m) -» T gT (G/T) 

depends only on gT, where r g is the map defined in (|3.7p . This induces an ori- 
entation preserving equivalence between E m and the tangent bundle T(G/T). So 
e(E m )([G/T}) = e(T(G/T))([G/T}) = X (G/T) = \W\. □ 

Theorem 3.4. We have e(E u / E V )([G/T}) = Cy. 

Proof. By Theorem 12.11 and Proposition ^. 31 

[Jo, Jo) 

So by Theorem 13 . 1 1 and Lemma l3.3[ we have 

e(E w )([G/T]) = <p{h){[G/T\) = ^^L(/ )([G/T]) 

[Jo, Jo) 

= { jTr!-e(E m )([G/T}) = { -/^-\W\ = c v . 
\jo,Jo) [fo,fo) 

□ 

4. Universal subspaces 

Given the representation of G in IX, we say that a linear subspace V of II is 
universal if every G-orbit in IX meets V. Let Gy be the stabilizer of V in G. 

Lemma 4.1. IfVis universal, then dim V > dim IX — dim G/Gy. 

Proof. Let Gxg v Vbe the vector bundle over G/Gy whose total space is the space 
of Gv-orbits in G x V with the Gv-action h ■ (g,v) = (gh^ 1 , p(h)(v)). Then the 
map F : G x V — * IX, F(g, v) = p(g)(v) reduces to a map F : G Xq v V — ► It. Since 
V is universal, F and F are surjective. So dim G/Gv + dimV = dim(G Xq v V) > 
dim IX. □ 

Assume that V is T-invariant, that is, T C Gy. By the above lemma, if V is 
universal, then dimV > dim IX — 2m, where m — | dim G/T. We say that V has 
optimal dimension if dim V = dim IX— 2m. In this case the characteristic polynomial 
fy of V has degree d = m. 

Theorem 4.2. Let p be a representation of G in a finite dimensional real vector 
space 11, and let V be a T-invariant subspace of XI such that XL T C V. Then fy ^ X 
implies that V is universal. In particular, if V has optimal dimension and Cy =/= 0, 
then V is universal. 

Proof. Let En, Ey be the vector bundles defined in Section 3. Every vector u € IX, 
viewed as a constant section of Eu, induces a smooth section s u of Eu/Ey. It is 
obvious that gT is a zero of s u if and only if u € p(g)(V), that is, p(g~ 1 )(u) e V. 
So V is universal if and only if for any u G IX, s u has a zero. This will be ensured 
if the Euler class e(Eu/Ey) is nonzero. 
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By Theorem EU e{E u /E v ) = <p(fv), where ip : S -> H*(G/T,R) is the homo- 
morphism (|2.3|) . By Theorem 12.11 the kernel of <p is X. So e(Ey_/Ev) ^ if and 
only if /v ^ X. This proves the main assertion. The particular case follows from 
Proposition ^. 31 □ 

In Theorem 14.21 when V has optimal dimension, the sufficient condition Gv ^ 
for the universality of V is fairly easy to check. If dimV > dimU — 2m, then 
deg/v < m. In this case, to determine whether fy £ T, one can use Theorem 
12.21 (5). Since 1 is generated by the basic generators Fi, . . . ,F r , one can also use 
the theory of Grobner bases (see [4]). But for many representations, this case can 
be reduced to the case of optimal dimension as the next proposition shows. (In a 
special case, this result was pointed out to us by J.-K. Yu.) 

Proposition 4.3. Suppose that the identity component of ker(p) is a torus, and 
suppose that there exists a G-orbit O in U such that dimO = dimG/ ker(p). IfV 
is a T -invariant subspace of U such that U T C V and /v ^ I, then there exists a 
T -invariant subspace V of 11 with optimal dimension such that Xi T C V C V and 
fv> $1. 

Proof. By replacing G with Gj ker(p) if necessary, we may assume that p is faithful. 
If V has optimal dimension, there is nothing to prove. So we assume that d = 
deg/v < m. Let Z = {(3 e t*|/v/9 € 1}- Then Z is a subspace of t*. We 
claim that 2 ^ t*. Indeed, if Z = t*, then /yt* C T, and then fvS m -d C T. 
Hence tp{f v )H 2 ( m ~ d \G /T,R) = tp(fv)<p(S m -d) = 0. But by the Poincare duality, 
the pairing H 2d (G/T, M) x H 2( - m - d ~> (G/T,R) -> H 2m (G/T,R) = R defined by 
(£1)62) !— * £1^2 is nondegenerate. As f(fv) ^ 0, we have a contradiction. 

We decompose V as V = U T (BiLi where each is 2-dimensional and 
T-invariant, and the T-action on Vi is equivalent to the real T-module associated 
to some x'i £ X(T). Since /y ^ X, V is universal. So n V 7^ and we can 
choose ueOflV. Then for any H S HiLi ker(Xi) and f g R, we have p(e tH )(v) = 
v. But dimO = dimG forces H = 0. So we have nf=i^ er (x«) — 0) that is, 
span{x' 1; . . . ,x'd'} = t*- Since Z 7^ t*, there exists io such that x- o ^ Z, that is, 
fvXi i I- Then, for V x = U T ffi l#lo V,, we have / Vl = JVx^ ^ X. Now the 
proposition follows by induction. □ 

Remark 4.1. By the principal orbit theorem (see e.g. [H]), the principal orbits 
have the maximal dimension and their union is an open and dense subset of 11. 
Hence there exists a G-orbit O with dimO = dimG/ker(p) if and only if the di- 
mension of the principal orbits is dimG/ker(p). Many representations satisfy this 
condition. For instance, the condition is satisfied by the complexified adjoint repre- 
sentation of G. Indeed, there are only finitely many real irreducible representations 
of G for which the condition fails. Moreover, in the case when G is simple, a com- 
plete list of real irreducible representations for which the condition fails is given 
in [16] . A similar list for complex representations (not necessarily irreducible) of 
complex simple Lie groups is given in [ID] . 

It is interesting to ask to what extent the converse of Theorem 14.21 holds. We 
now prove a special case. Further discussion will be given in Section 6. 

Suppose that V has optimal dimension, and let W be a T-invariant subspace of 
It complementary to V. Let g and m be as before. For every v G V, we define a 
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map ipv ■ na — > W by 

MX) = -Pwdp(X)(v), (4.1) 
where Pw is the projection of II on W along V, and dp : g — > EndCU) is the 
differential of p. Note that since V has optimal dimension, we have dimm = dim"W. 
Let det (?/>„) be the determinant of the matrix of ip v with respect to the basis (I3.6[) 
of m and the basis 

{wi,JyfWi,...,w m ,Jww m } (4.2) 
of W (see (I3.3|) ). The map v i— » det(ip v ) is a real polynomial function on V. 

Theorem 4.4. Lei £/ie notation be as above. Suppose that det(tp v ) does not take 
both positive and negative values on V. Then the following statements are equiva- 
lent: 

(1) Cv ? 0; 

(2) V is universal; 

(3) det(t() v ) is not identically zero. 

Before proving the theorem, we first make some preparations. Recall from the 
proof of Theorem [42] that we associated to each sella section s u of the oriented 
bundle Eu/Ey over G/T, and that a point gT G G/T lies in the zero locus Z(s u ) 
of s u if and only if p(g~ 1 )(u) G V. We denote the G-orbit of u by O u . 

Lemma 4.5. Let u S It, x = gT G Z(s u ) and v = p(g~ 1 )(u). Then 

(1) ind x (s u ) = sgn(V^); 

(2) O u rh„ V s u (f\ x s <=> det(ip v ) ^ 0. Ira particular, O u (f\ V if and only if 

s u rti s Q . 

Proof. (1). Since Eu/Ey is equivalent to I?w, s„ can be viewed as a map s u : 
G/T — > It and can be expressed as 

= P p ( s )( W )(w) = p(.g)P w p(.g _1 )(M), 

where P p ( g )(w) is the projection of It onto p(g)(W) along p(£f)(V). Since x G Z(s u ), 
we can view the tangent map (ds u )x as a map from T X (G/T) into p(g)(W). Let r ff 
be the map defined in (|3.7jl . Then we have 

s u o r g {X) = p(g)p(e x )P w p(e- x )p(g~ 1 )(u) = p(g)e d ^ P we - d ^ x \v). 

Since v G V, 

(d* u ) x o (dT fl ) (X) = p( 5 )(rfp(X)P w («) - Pwd P (X)(v)) 

= - p(g)P w dp(X)(v) = p{g)MX), 

that is, (ds u ) x (dT g )o = p(g) o ?/v Since both (dr s )o and p(g) : W — > p(g)(W) are 
invertible and preserve the orientation, we have ind a; (s tl ) = sgn((ds„) x ) = sgn(^>„). 

(2). The assertion s u &\ x s o ^ det(tp v ) ^ follows directly from (1), and the 
assertion O u fr\ v V det(ip v ) ^ is obvious from the definition oiip v . □ 

We say that a flag gT G G/T sends u into V if gT G Z(s u ). By the above lemma, 
if O u rtl V, then the number 

JV(u,V) = #Z(su) 

of flags sending it into V is finite. By the transversality theorem (see [13|), the 
vectors u G It for which O u rtl V form an open and dense subset of U. So N(u,V) 
is finite for generic u G It. In general, N(u, V) is not constant even for generic u. 
But this is the case if the condition of Theorem 14 . 41 holds . 
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Lemma 4.6. Under the condition of Theorem \4-4\ for u G U with O u ftl V, we have 
N(u,V) = \Cv\. 

Proof. Without loss of generality, we may assume that det(V>„) > for every v £ V. 
Since O u ftl V, for every x = gT £ Z(s u ) and v — p(g^ 1 )(u) £ V, we have det(ip v ) ^ 
0, which implies that det(ip v ) > 0. So for such x and v, ind x (s u ) = sgn(ip v ) = 1. 
By Theorem 13. 4[ we have 

N{u,V) = #Z{s u )= ind *( s «) 

xEZ(s u ) 

=I{s u , s ) = e{E u /Ev)([G/T}) = C v . 

□ 

Proof of Theorem\J^ "(1) => (2)". This follows directly from Theorem[I2] 

"(2) (3)". Since the map G X U — > IC, i— > p(g)(u) is transversal to V, 

by the transversality theorem, there exists u € U such that O u rtl V. Since V is 
universal, O u flV ^ f). So we may choose v £ O u D V and we have O u ftl t , V. So 
det(Vv) ^ 0. 

"(3) =^ (1)". Choose w e V such that det(Vv ) 7^ 0. Then O vo rh„ V. Consider 
the map F : G x V — * It, F(g, w) = p(g)(v). Then (dF)( ei „ ) is surjective. Hence the 
image Im(i 7 ') of -F contains an open neighborhood of vo in U. By the transversality 
theorem, we can choose u £ Im(F) such that O u ftl V. Since u £ Im(F), O u DV 7^ 0. 
So Z(s u ) is not empty. By LemmalHl \C V \ = N{u,V) > 0. □ 

Now we give an example for which the assumption of Theorem 14.41 holds . Let mc 
be the complexification of m, and let n ± = Qe $± 0a, where $~ = — $ + . Then 
mc = n + © n~ . Recall that W can be equipped with the complex structure Jw as 
in (|3.2p . Then the map ip v : m — > W extends uniquely to a complex linear map 
(ip v )c ■ m c -> (W, J w ) for all v G V. 

Remark 4.2. As in Remark 13. 1[ if U admits a complex structure such that the 
G-action is complex linear, and if V, W are complex subspaces of U, then Jw can 
be chosen to be the same as the original complex structure on W. In this case, the 
homomorphism dp : g — > Endc(U) extends uniquely to a complex homomorphism 
(dp)c '■ 0c - * Endc(U), and we have 

(^)cW = --PwWcW(») 

for v G V and X G mc- 

Corollary 4.7. Lei £/ie notation be as above. Suppose that, with respect to the 
complex structure Jw, t/iere is a complex decomposition W = W + 0W" suc/i £/ia£ 
( - 0i))c(n ± ) C "W^ for all v £ V. Then V is universal if and only if Cv 7^ 0. 
Furthermore, if u £ 11 and O u ftl V, then N(u,V) = \Cy\. 

Proof. By Theorem 14.41 and Lemma T4.61 it suffices to show that det(ipv) does not 
take both positive and negative values on V. 

Let J w be the complex structure on W defined by 

(j{ v (w) = Jw{w), w£W+; 

l4W = ^wW, w£W-. 
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Then for a € $ + , 

ipvJ m {X a - X_ a ) =ip v (V^l(X a + X- a )} 

={i>v)c{V=lX a ) + {ip v ) c (V=lX- a ) 
=Jw(ipv)c(X a ) + Jw(^v)c(X- a ) 
=Jw(^v)c(X a ) - J^ v ) c (X_ a ) 

=Jw'4'v(x a — X- a ), 

ip v J m (V^l(X a + X- a )) = - 1p v (X a - X_ a ) 

= JyjJ{y'lp v (X a — X- a ) 
= J , W' l PvJm{X a — X— a ) 

=j^ v (V=i(x a + X- a )). 

So the map if> v : m — > W is complex linear with respect to the complex struc- 
tures J m and J(y. Hence with respect to the basis p.6[) of m and the basis 
{w[, J^w[, . . . , w' m , J(v w m} °f t ne matrix of ip v has nonnegative determinant, 
where {w' 1: . . . , w' m } is any complex basis of (W, J{y). So with respect to the basis 
dS^D of m and the basis ||II2J) of W, either det(^) > for all v £ V, or det(^) < 
for all veV. □ 

5. Generalizations of Schur's triangularization theorem 

The classical Schur triangularization theorem asserts that for the complexified 
adjoint representation of U(n) in M(n, C), the subspace of M(n, C) consisting of all 
upper (or lower) triangular matrices is universal. We generalize this theorem to a 
whole class of subspaces of M(n, C) by using the results in the previous sections. 
We shall also obtain similar generalizations for the conjugation representations of 
Sp(n) and SO(n) in M(n,H) and M(n, R), respectively. To unify the results, we 
introduce the following definitions. 

Definition 5.1. Let n be a positive integer. 

(1) An n x n zero pattern is a set 

I C{(i,j)\i,j €{l,...,n},i^j} 
of cardinality n(n — 1) /2. 

(2) The characteristic polynomial Xj of an n x n zero pattern I is 

Xi(x) = Yl i^i-Xj) eR[xi,...,x n ]. 

(3) The characteristic number of I is Ci — (Aj,Ao), where Ao is the polynomial 
(|2.6|) . and (•, •) is the inner product on R[xi, . . . ,x n ] for which the set of monomials 
{a;™ 1 • • • is an orthonormal basis. 

It is obvious that Ao is the characteristic polynomial of the upper triangular zero 
pattern 

Io = {(i,j)\l<i<3 <n}, 
and that Cj = (Ao, Ao) = n\. 

We first consider the complex case. For A 6 M(n, C), let Ay denote the (i,j)-th 
entry of A. 
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Theorem 5.1. Consider the complexified adjoint representation ofU(n) in M(n, C). 
Let I be an n x n zero pattern. If C'i ^ 7 then the subspace 

Mj(n,C) = {Ae M(ra,C)|A y = for (i,j) e 1} 

is universal. 

Proof. We keep the notation from Example 12.11 For (i,j) G /, let 

Wy ={ie M(n,C)\A kl = if (fe,/) ^ (»,i)}. 

Then Wy is T-invariant and T acts on Wy via the character xt — Xj (identified 
with t i — ► titj 1 ). The subspace "W = ®^ of M(n, C) is complementary to 

V = M/(rt, C). Note that V has optimal dimension. By Remark |3. 11 we have 

h(x) = Yl ( X i ~ x i) = Aj(:z), 

(i,3)el 



and 



n (fv, fo) i c i (Aj, A ) 



(/oj/o) (AojAq) 
So by Theorem 14. 2[ V is universal. □ 



Note that M/ (n, C) consists of all lower triangular matrices. So if / = I , the 
above theorem reduces to the classical Schur triangularization theorem. 

We say that a zero pattern / is bitriangular if («o> Jo) G ^ an d (jo ~ *o)(*o — jo + 
j — i) > imply that G /. If zo < jo (resp. io > jo), this condition says that 
G I whenever j — i > jo — io (resp. i — j > io — jo)- Using Corollary |4.71 we 
can prove the following theorem. 

Theorem 5.2. Let I be a bitriangular zero pattern. Then M/(n, C) is universal if 
and only if Ci ^ 0. Furthermore, if the U(n) -orbit of A G M(n, C) is transversal 
to M/(n,C), i/ien AT(A, M/(n, C)) = \C I \. 

Proof. In the notation of Corollarv l4.71 n + (resp. n _ ) is the set of all strictly upper 
(resp. lower) triangular matrices in M(rt, C). Let V = M/(n, C), and let W be the 
subspace defined in the proof of Theorem OandW ± = Wnn ± By Remark EU 
for v G V, we have (ip v )c(X) = Pw([v,X]), X G mc- Since I is bitriangular, it is 
easy to see that PwQV,^]) C W ± . So (V't,)c(n ± ) C W ± for all v G V, and the 
theorem follows from Corollarv l4.7l □ 



Remark 5.1. For the complexified adjoint representation of U(n) in M(n, C), the 
number N(A, M/(n, C)) of flags sending A into M/(n, C) can be interpreted as fol- 
lows. By viewing A as an operator on C™ and a flag as an ordered n-tuple (li,. .., l n ) 
of mutually orthogonal complex lines in C™, N(A, Mj(n, C)) is the number of flags 
(1%, . . . , l n ) such that with respect to an ordered basis {ei, . . . , e„} of C n with e, G h, 
the matrix of A is in Mj(n, C). 

Now we consider the quaternionic case. For a matrix A G M(n, H), let Ay denote 
the (i, j)-th entry of A. 

Theorem 5.3. Consider the conjugation representation of Sp(n) in M(n, H). Let 
I be an n x n zero pattern. If Cj ^ 0, then the subspace 

M/(n,H) = {A G M(n,H)|Ay = /or (i, j) G /} 

is universal. 
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Proof. We keep the notation from Example 12.21 Let V be the subspace consisting 
of all matrices A £ M/(n, H) such that A it e C for 1 < i < n. For e I, let 

Wy ={Ae M(n,M)\A kl = if (k,l) ? 

and let 

W, = {A e M(n, M)\Au G jC and A kl = for (fc, I) ^ (i, i)} 
for 1 < i < n. Then the subspace 

W= ( W, 

is complementary to V in M(n, H). It is easy to see that each T- invariant subspace 
Wij can be decomposed as the direct sum of two T-irreducible subspaces and, with 
respect to suitable orientations, they are equivalent to the characters Xi + Xj and 
Xi — Xj of T. Moreover, each Wj is T-irreducible and, with respect to suitable 
orientation, it is equivalent to the character 2xi of T. So we have 

n n 

(i,i)e/ <=i t=i 

Note that V has optimal dimension, and the endomorphism of R[xi, . . . , x n ] sending 
f(xi, . . . , x n ) to /(xf, . . . ,a£) n"=i ^ is isometric. So 

C V = ^t|(Z/2Z)" x S„| - -^442"n! = 2™C I . 
\J0, Jo) (A ,Ao; 

By Theorem 14. 21 if Cj 7^ then V is universal, and a fortiori Mj(n, H) is universal. 

□ 

For the real case, we first consider the case of M(2n,R). We partition a matrix 
A G M(2n, R) into 2x2 blocks and denote its (z, j)-th block entry by a,ij(A). 

Theorem 5.4. Consider the conjugation representation of SO(2n) in M(2n, R). 
Let I be an n x n zero pattern. If C'i 7^ 0, then the subspace 

M/(2n,R) = {A e M(2n,R)\ aij (A) = for E 1} 

is universal. 

Proof. We keep the notation from Example 12.31 For (i,j) € /, let 

={Ae M(2n,R)\a k i(A) = if {k,l) ^ 

Then W = (l j)e/ W tJ is a subspace of M(2ra, R) complementary to V = M/(2n, R). 
As in the proof of Theorem I5.3L each decomposes as the direct sum of two 
T-irreducible subspaces, and with suitable orientations, they are orientation pre- 
serving equivalent to the characters Xi + Xj and a;, — Xj of T, respectively. So we 
have 

h(x) = Yi ( x i ~ x j) = X i( x i> ■ ■ • ) x n)- 
(i,j')ei 

Note that V has optimal dimension, and the endomorphism of R[xi, . . . , x n ] sending 
f(xi, . . . , x n ) to /(xf , . . . , is isometric. So 

c v = ^^|(z/2Z)"- 1 x s B | = ^44 2 «-V = a"- 1 ^. 
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By Theorem [OJ if C/ ^ then V/ is universal. □ 

For the case of M(2n + 1,K), we partition a matrix A e M(2n+ 1,R) into blocks 
ciij(A) of size (2 — <5i.„+i) x (2 — 5j >n +\), where 5ij is the Kronecker symbol. 

Theorem 5.5. Consider the conjugation representation of SO(2n + 1) in M(2n + 
1, K). Le£ J oe an (n + 1) x (n + 1) zero pattern such that exactly one of (i,n + 1) 
and (n + l,i) is m J for any i < n, and let I = £ J\i,j < n}. If Cj ^ 0, 

then the subspace 

Mj(2n+1,K) = {A £ M(2n + l,R)\aij(A) = for £ J} 

is universal. 

Proof. The subspace 

W = {A £ M(2n + l,R)\aij(A) = for £ J} 

of M(2n + 1, R) is complementary to V = Mj(2n + 1, R). By choosing a decompo- 
sition of W and suitable orientations on the direct summands, we compute 

n n 
(i,j)£l i=l i=l 

and 

Cv = ^t|(Z/2Z)" x S n \ = -^442"n! = 2 n Cj. 
[fa, Jo) \A ,A ; 

So if Ci ^ 0, then by Theorem |OJ V is universal. □ 



Remark 5.2. It is easy to see that the condition of Proposition 14. 31 is satisfied for 
all representations considered in this section. 

We say that an n x n zero pattern / is simple if / contains exactly one of (i, j) 
and (J, i) for any i, j G {1, . . . , n} with i =/= j. 

Theorem 5.6. Let I (resp. J) be a simple n x n (resp. (n + 1) x (n + 1)J zero 
pattern. Then in the context of Theorem \ 5.1\ ( resp. 15.31 \5.4\ 1 5. 5\) the subspace 
M/(ra,C) (resp. M 7 (n,H), M 7 (2n,M), Mj(2n + 1,M)J is universal. 

Proof. We have 

A/(x) = I [ (xi-Xj) = ± Yi (xi- Xj) =±Xo(x). 

So Cx = (A/,Ao) = ±(Ao,Ao) = ±nl, and the assertion follows from the above 
theorems. □ 

The problem of universality of the subspace in the following example was raised 
in [7] but seems difficult to prove by other methods. 

Example 5.1. Let n = 3. Consider the cyclic zero pattern 

/= {(1,3), (2,1), (3, 2)}. 

Since I is a simple zero pattern, M/(3, C) is universal with respect to the conjugation 
representation of [/ (3) in M(3, C). □ 
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Example 5.2 ( 23, [5]). Let n = 4. Consider the bitriangular zero pattern 

I ={(1,3), (1,4), (2, 4), (3,1), (4,1), (4, 2)}. 

Its characteristic polynomial is 

\i{x) = — (aci - x 3 ) 2 (xi - x 4 ) 2 (x 2 - Xif, 

and then Cj = (A/,Ao) = —12. So M/(4, C) is universal with respect to the 
conjugation representation of U{4) in M(4, C). Since / is bitriangular, for A £ 
M(n,C) with O a rh Mj(4,C) we have N(A,Mi(A, C)) = 12. □ 

Using the same method, it is easy to generalize Theorems l5.2l and l5.6l to arbitrary 
complex representations of any connected compact Lie group. We state the result 
for the complexified adjoint representation below and leave the details to the reader. 

Theorem 5.7. Consider the complexified adjoint representation of a connected 
compact Lie group G in Qc- Let ^ be a subset of the root system $ such that 
|*| = |* + l> and Zef V = tc©© Qe$vt a Q . 

(1) If § = * U (-*), then V is universal; 

(2) If a € W H and f3 £ (with the same choice of signs) imply that a + (3 </ 
$ \ then V is universal if and only if Gv 7^ 0. Furthermore, if X G 0c and 
O x rh V, then N(X,V) = \C V \. 

6. Concluding remark 

In the previous sections, we assumed that T C Gv, and then constructed the 
vector bundles Ey$ = Eu/Ey over G/T, and reduced the question of universality 
of V to the existence of zeros of certain sections of Ey$ . A sufficient condition for 
this is that the Euler class e(Eyj) is nonzero. 

This method remains valid if we replace T by any closed subgroup H of Gv- 
The condition that T C Gv can be also dropped. More precisely, for any such H, 
we let W be a Gv-invariant subspace of II complementary to V, and construct the 
subbundles Eh,v and Eh,w of the trivial bundle Eh,u = G/H x U whose fibers 
at g H are p(g)(V) and p(g)(W), respectively. Similarly to Theorem 14.21 if certain 
characteristic class o(E H yj) (e.g. Euler, top Stiefel- Whitney, or top Chern class) 
of Efj.w is nonzero, which ensures that every smooth section of Sjj w nas a zero, 
then V is universal. For example, if H preserves an orientation on W, then Eh,w 
is orientable and we can use the Euler class. 

If Hi C H2 are two closed subgroups of Gv , then there is a natural quotient map 
q : G/Hi — > G/H2, and it is easy to see that Eh x ,w — q*(EH 2 ,w)- Hence we have 
o(Eh 1 ,w) = q* ( (Eh 2 ,w))- If q* is not injective, it may happen that o(Ejj 1 ,w) = 
but o(Eh 2 ,w) 7^ 0. So by using a larger subgroup H of Gv, we may get stronger 
sufficient condition for the universality of V. 

An interesting case occurs when dimV = dimU — dim G/Gv and T C Gv- We 
further assume that T is the identity component of Gv- Then Gy/T is a subgroup 
of the Weyl group W . To avoid the trouble with non-orientability, we assume that 
11 admits a complex structure such that the G-action is complex linear, and that 
V and W are complex subspaces. Then we can use the top Chern class (which 
is equal to the Euler class of the underlying real bundle). We choose H± = T, 
H 2 = G v . Then dim G/T = dim G/G v = 2m, and c t0 p(£ T:W ) € H 2m (G/T,Z), 
Cto P (EG v ,w) € H 2m (G/Gv,Z). It is easy to prove that G/Gv is non-orientable if 
and only if Gv /T contains an odd element of W. From algebraic topology, if G/Gv 
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is orientable, then H 2m (G/G v ,%) = % and q* : H 2m {G/G v ,%) -> H 2m {G/T,Z) 
is injective. So we obtain the same information from the top Chern classes by 
using T and Gy. But if G/G v is non-orientable, then H 2m (G/G v , Z) = Z/2Z and 
<f : H 2m (G/G v ,Z) -> H 2rn {G/T,Z) is equal to zero. In this case we always have 
Cto P (-Er,w) = q*(c t op(£ , Gv,w)) = 0, but it may happen that c top (E GvtW ) ^ 0, and 
if this is the case, we get more information than by using T and we can prove the 
universality of V. We demonstrate this in the following simple example. 

Consider the complexified adjoint representation of G = SU(2) in IX = st(2, C). 

Let V = | r *U. Then G v = N G (T). Let W = | f J _°J |o e c|. Since 

Gv/T = W contains an odd element, e(Er,w) = 0. Indeed, Et,w is the trivial 
complex line bundle over G/T = S 2 . But it is easy to see that Eo v ,w> as a 
complex line bundle over G/Gv = MP 2 , is equivalent to the complcxification of 
the tautological real line bundle over RP 2 , which has nonzero first Chern class. So 
Ctop(-E , G v ,w) 7^ and V is universal. 

The above consideration motivates us to ask the following question. 

Question. Suppose that dimV = dim IX — dim G/Gv cind T C Gv- Does the 
universality of V imply that certain obstruction class of Eg v ,Wi f or the existence 
of non-vanishing sections, is nonzero? 

We remark that if the condition dimV = dim II — dim G/Gv or T C Gv is 
dropped, then in general the answer is negative. This can be seen from the following 
counter-examples. 

For the case of T <f_ Gv, consider the representation e %e i— ► diag(e* e , e 2ie ) of U{\) 
in C 2 . We view C 2 as a real vector space, and let V = {(a, b) £ <C 2 |Re(a + b) = 0}. 
Then Gv is trivial, and Eg v ,w is a trivial real line bundle over U(l). But it is easy 
to see that V is universal. 

For the case of dim V > dim IX — dim G / Gv , we can consider the complexified 
adjoint representation of any semisimple compact group G of rank greater than 1 
in 2c- Let X be a regular element in t, let W = CX, and let V = W with respect 
to the Killing form of gc- Then Gv = T, and the above inequality holds. It is easy 
to see that Ec v ,w is a trivial complex line bundle over G/T. But since V contains 
the sum of all root spaces in gc, V is universal by [8], Theorem 3.4. 

References 

[1] I. N. Bernstein, I. M. Gel'fand, S. I. Gel'fand, Schubert cells and cohomology of the spaces 

G/P, Russian Math. Surveys, 28 (1973), no. 3, 1-26. 
[2] A. Borel, Sur la cohomologie des espaces fibres principaux et des espaces homogenes de 

groupes de Lie compacts, Ann. of Math. (2), 57 (1953), 115-207. 
[3] R. Bott, L. Tu, Differential forms in algebraic topology, Springer- Verlag, New York-Berlin, 

1982. 

[4] D. Cox, J. Little, D. O'Shea, Ideals, varieties, and algorithms: an introduction to computa- 
tional algebraic geometry and commutative algebra, Springer, New York, 2007. 

[5] K.R. Davidson, D.Z. Dokovic, Tridiagonal forms in low dimensions, Linear Algebra Appl., 
407 (2005), 169-188. 

[6] D.Z. Dokovic, Universal zero patterns for simultaneous similarity of several matrices, Oper. 

Matrices, 1 (2007), no. 1, 113-119. 
[7] D.Z. Dokovic, C.R. Johnson, Unitarily achievable zero patterns and traces of words in A and 

A*, Linear Algebra Appl., 421 (2007), no. 1, 63-68. 
[8] D.Z. Dokovic, T.-Y. Tarn, Some questions about semisimple Lie groups originating in matrix 

theory, Canad. Math. Bull., 46 (2003), no. 3, 332-343. 



20 



J. AN AND D.Z. BOKOVIC 



[9] J. Dupont, Curvature and characteristic classes, Lecture Notes in Mathematics, Vol. 640, 

Springer- Vcrlag, Berlin-New York, 1978. 
[10] A. G. Elashvili, Canonical form and stationary subalgebras of points of general position for 

simple linear Lie groups, Funct. Anal. Appl., 6 (1972), no. 1, 44-53. 
[11] W. Fulton, Young tableaux, London Mathematical Society Student Texts, 35, Cambridge 

University Press, Cambridge, 1997. 
[12] J. Galindo, P. de la Harpe, T. Vust, Two observations on irreducible representations of 

groups, J. Lie Theory, 12 (2002), no. 2, 535-538. 
[13] V. Guillcmin, A. Pollack, Differential topology, Prentice-Hall, Inc., Englewood Cliffs, N.J., 

1974. 

[14] S. Helgason, Groups and geometric analysis, American Mathematical Society, Providence, 
RI, 2000. 

[15] J. Holbrook, J. -P. Schoch, Moving zeros among matrices, with an appendix by T. Kosir and 

B.A. Sethuraman, Linear Algebra Appl., 424 (2007), no. 1, 83-95. 
[16] W.-C. Hsiang, W.-Y. Hsiang, Differentiable actions of compact connected classical groups, 

II, Ann. of Math. (2), 92 (1970), 189-223. 
[17] R. Kane, Reflection groups and invariant theory, Springer- Verlag, New York, 2001. 
[18] K. Kawakubo, The theory of transformation groups, Oxford University Press, New York, 

1991. 

[19] R. Kiehl, R. Weissaucr, Weil conjectures, perverse sheaves and l-adic Fourier transform, 
Springer- Vcrlag, Berlin, 2001. 

[20] H. Kraft, N.R. Wallach, On the separation property of orbits in representation spaces, J. 
Algebra, 258 (2002), no. 1, 228-254. 

[21] M. Mimura, H. Toda, Topology of Lie groups, I, II, American Mathematical Society, Provi- 
dence, RI, 1991. 

[22] J. Milnor, J. Stasheff, Characteristic classes, Princeton University Press, Princeton, 1974. 
[23] V. Pati, Unitary tridiagonalization in M(4, C), Proc. Indian Acad. Sci. Math. Sci., Ill (2001), 
no. 4, 381-397. 

[24] M. Reeder, On the cohomology of compact Lie groups, Enseign. Math. (2), 41 (1995), no. 3-4, 
181-200. 

[25] I. Schur, Uber die charakteristischen Wurzeln einer linearen Substitution mit einer Anwen- 
dung auf die Theorie der Integralgleichungen, Math. Ann., 66 (1909), no. 4, 488—510. See 
also Gesammelte Abhandlungcn, Band I, 272-294. 

[26] L. Smith, Polynomial invariants of finite groups, A. K. Peters, Ltd., Wellesley, MA, 1995. 

[27] J.-K. Yu, On the motive and cohomology of a reductive group, preprint. 

Department of Pure Mathematics, University of Waterloo, Waterloo, Ontario, N2L 
3G1, Canada 

Current address: LMAM, School of Mathematical Sciences, Peking University, Bei- 
jing, 100871, China 

E-mail address: anjinpengagmail.com 

Department of Pure Mathematics, University of Waterloo, Waterloo, Ontario, N2L 
3G1, Canada 

E-mail address: djokovic@uwaterloo.ca 



